Participants: Cristiano Ciaccio, Alessio Miaschi, Michele Papucci, Andrea Pedrotti, Giovanni Puccetti.
Classifiers on the dataset, function really well. It was tested:
- RoBERTa: It works really well, close to perfection. Various different settings were used:
- Dataset Size: 100k vs 10k. No difference.
- Contest Windows: 8 vs 16 vs 32 vs 256 vs 512. Basically no difference. 8 tokens performs the worst going at around 80% accuracy. The place where the 8 tokens are taken (head, middle, tail) doesn’t seem to matter.
- SVMs. They work really best, from 95 to 98% F-Scores. Various settings:
- Lexical SVM using TF-IDF of words as features. 95% F-Score;
- Profiling SVM trained with Profiling-UD features. 98% F-Score;
- This was further trained removing the top 10 features (by coefs) at each iteration. Performances drop steadily but with 16/154 feature remaining still goes at around 60% F-Score.
Most frequent words analysis reveal a very different distribution between real and synth (fakes). That is probably why SVM with TF-IDF works so good.
Why is the task that easy?
Maybe the problem is xsum, and BBC article are easy to target. However empirically humans have a much harder time.
It seems to be really well-splitted and the task may be that easy for this precise reasons (T-sne visualization).
What to do from now.
We start by using the llama3-8b
Re-generating the sentences from the headlines of xsum, and cutting them off at the same length to control that variable.
Then, for prompting with verbalized features:
- We look at decision tree cutoffs to verbalize the feature that can easily be express in Natural Language
For, DPO: - We do a run training the model using DPO with the real data, aka preferring the o.g. article vs the synth articles.